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Abstract- Complex human activity recognition suffers from ambiguity of interpretation problem. A 
novel neutrosophic formal concept analysis method has been proposed to quantify non-determinism 
leading to ambiguity of interpretation and utilize it in activity recognition. The method works by 
penalizing performance of non-deterministic activities and rewarding the deterministic ones. Thus, 
non- deterministic activities are identified during testing due to significantly reduced performance and 
contexts can be redesigned to improve their description. The proposed method has been implemented 
on benchmark dataset having both types of activities. Our approach successfidly identified non- 
determinism in activities description without compromising recognition performance of deterministic 
activities. It has also been shown that other approaches fail to identify non deterministic activities. 
Overall accuracy of activity recognition of our approach was comparable to other approaches. 

Index terms: Sensors Data Streams, Concept Lattice, Neutrosophic Logic, Situation Inference, Activity 
Recognition. 
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I. INTRODUCTION 

Regular performance of Activities of Daily Living (ADL) has been established as major indicator 
of wellbeing in humans [1], Video based monitoring with miniature cameras provide rich 
information about the environment being monitored. They are however, not preferred in 
monitoring ADLs because of privacy and ethical issues. Unobtrusive sensors are more easily 
acceptable by human beings as they don’t appear to directly invade their privacy. This has given 
rise to significant increase in research reported on sensor based Activity Recognition (AR) [2]. 

In this work, an activity has been described by a set of symbolic information called “contexts”, 
obtained from real time raw sensor data processing [3]. Significant success has been achieved in 
extracting low level contexts from sensor data [4] . However, machine based AR from Contextual 
Information (Cl) is still challenging due to several reasons [5]. 1) Humans rarely follow a fixed 
sequence while doing any activity. Same sequence of actions in an activity is further unlikely 
when different humans do same activity. 2) More than one activity may take place at same place 
and within same time frame. Location and time thus can’t distinguish these activities. 3) Humans 
may interleave several activities together. For example, cooking and answering phone call may 
be interleaved. 4) Contextual description of activities can be ambiguous when same contexts are 
present in different activities. 

A formal concept lattice based solution approach to handle first three problems has been 
suggested by us in [6]. In this paper, we focus on addressing the last challenge. Contexts, as 
perceived in this work, are middle level abstractions obtained as semantic interpretation of raw 
numeric sensor data. For example, data from relevant sensors for the context left_arm_action can 
be interpreted as move, release, open, close, cut, bite and stir etc. Activities can be described as 
co-occurrence of specific unique set of values of various such contexts. However, on observing 
real activity datasets it is found that combination of context values describing some activities may 
not be unique. This leads to non-determinism in contextual description of that activity. Non- 
determinism occurs either due to small number of possible contexts or due to inherent variation 
in the way person/s performs an activity each time. For example, left_hand_action - ‘move’ can 
be present in many activities done by a person at home. If at least one distinguishing context 
value is not present than the activity description becomes ambiguous. 
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In this paper, we propose a new neutrosophic formal concept analysis based AR model to handle 
the challenge of ambiguity in context based interpretation of activities. The method works by 
penalizing performance of non-deterministic activities and rewarding the deterministic ones. 
Thus, non-deterministic activities are identified during testing due to significantly reduced 
performance and contexts can be redesigned to improve their description. A benchmark data set 
of mid- level abstractions and corresponding activities has been used for validation [7]. On 
testing, activity recognition method successfully recognized well defined activities and identified 
non-determinism in description of other activities. Upon identification, more number and types of 
sensors can be deployed to deterministically define those activities. 

The rest of paper is organized as follows. In the next Section, features and drawbacks of 
existing activity recognition methods have been discussed. In Third Section, preliminaries of 
concept lattice and neutrosophic logic are described. Neutrosophic concept lattice based activity 
model has been proposed in Section 4. Method of utilizing the model for activity recognition has 
been described in the same section. Proposed methods were validated on third party datasets. 
Description of the datasets, model obtained for the dataset and other validations have been 
presented in Section 5. The paper is finally concluded in Section 6. 

II. RELATED WORK 

There has been recent research interest in automatic activity recognition in pervasive computing 
environments [2]. Existing literature was surveyed to explore related existing methods. For 
activity model creation, foremost requirement is to model relationships between contexts and 
activities quantitatively. One class of methods proposed in literature use annotated datasets of 
actual AR environments to train and learn probabilistic or statistical activity models based on 
machine learning techniques [8- 12]. Mahmod et al [8] examined the ability of recurrent neural 
networks for modelling sensory data to ADLs of an occupant in inhabited intelligent 
environments. Behavioral model of each occupant was built using this technique. The model 
predicted future activities and expected occupancy. 

Hidden Markov Model has been applied for predicting activity series from low rate sensor signals 
obtained from wearable accelerometer sensors in [9]. Dynamic Bayesian Networks (DBN) has 
been used in Assisted Cognition project [10] to provide directional guidance to a user navigating 
through a city. This system uses a three-level hierarchical Markov model represented as a DBN 
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to infer user’s activities from GPS sensor readings. Movement patterns based on GPS localization 
signals are translated into a probabilistic model using unsupervised learning. Brdiczka et al. [11] 
have recently used Support Vector Machine (SVM) to locate and recognize social interactions of 
subjects from multiple sensors, including video and audio. High precision and recall was 
obtained in identifying low-level interactions like shaking hands, touching, pushing, and kicking. 
Ravi et al. investigated multiple base-level classifiers namely decision tables, decision trees, K- 
NNs, SVM, and Naive Bayes applied through a variety of techniques such as boosting, bagging, 
voting, cascading, and stacking on worn accelerometer data[12]. Boosted SVM was shown to 
perform best for difficult settings in recognizing activities like standing, walking, running, going 
up/down stairs etc. 

The task of AR with respect to establishing wellness of monitored person has been discussed in 
[13 - 15]. Similar to our assumption, Suryadevara et al assume activities to be some sequence of 
sub activities derivable from sensors [13]. The sub activities have been determined using naive 
bayes model. Maximum likelihood estimate have been used to estimate activity from the given 
sequence of sub- activities within a given time frame. Similarly, emotional state of person was 
determined from other related sensors. Wellness was computed from these two by defining and 
calculating wellness indices. Such an indoor tracking system has been implemented in [14] and 
has been shown to determine irregular health conditions correctly. The wellness indices based on 
usage duration and time of household appliances were designed in [15]. Use of wellness indices 
helps in increasing the objectivity of all AR methods. There is still scope of research in defining 
custom wellness indices using individual parameters. 

Apart from the discussed data-driven machine learning based methods, recently, ontology based 
formalization of activity models has also gained popularity. Riboni and Bettini [16] investigated 
the use of activity ontologies to model, represent, and reason complex activities using rule 
representation and rule-based reasoning of Web Ontology Language (OWL). 

Generic ontologies for situations were defined by making use of intrinsic and extrinsic contexts. 
Use of existing and new ontologies has been suggested to map multiple terms used across 
methods to defined contexts and activities [17]. Ontology based approaches address problem of 
contextual non-determinism to some extent by grouping them into a super activity say “Kitchen 
activity” and present that as result of AR. However, such answer is not appreciable when there is 
a need to identify each ADL separately. 
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Ontology based activity recognition methods are semantically more clear and understandable for 
human beings than data driven methods. However, experimental evaluation of these techniques 
has been limited and their actual effectiveness is still unknown. To the best of our knowledge, 
none of the existing methods addresses ambiguity of interpretation problem in composition of 
activities in terms of high-level context information. 

Use of fuzzy logic to capture uncertain constituent information of activities has been proposed by 
few researchers. Fuzzy Cognitive Maps (FCM) for describing behaviour of a system in terms of 
concepts; each one representing an entity, a state, a variable or a characteristic of the system has 
been proposed in [18]. FCMs indicated the relationships between the environmental variables and 
the emotions. What-if simulations and forecasting emotions according to previous environmental 
conditions were shown. Fuzzy logic based approach measures uncertainties as degree of truthness 
of membership of contexts in a situation. However, the problem of non-determinism mentioned 
in Section 1 cannot be handled by fuzzy logic. An activity modeling method based on descriptive 
Formal Concept Analysis (FCA) has been proposed by us in [6]. To handle non-determinism, 
FCA has been enhanced with neutrosophic logic and comparisons are made with conventional 
fuzzy concept lattices. 

Neutrosophic logic has been a topic of interest among mathematicians but has not been applied 
much in applications [19]. Use of neutrosophic logic in classification problems has been limited. 
One significant work towards use of neutrosophic logic has been presented in [20], where 
neutrosophic logic based extension of fuzzy classifier has been designed. Although, fuzzy 
concept lattices have been worked upon by many researchers, this is the first attempt to further 
generalize concept lattices with neutrosophic logic. 

III. NEUTROSOPHIC LOGIC AND FORMAL CONCEPT ANALYSIS 

In this section, preliminaries of neutrosophic logic and neutrosophic formal concept analysis have 
been defined. 

a. Preliminaries of Neutrosophic Logic 

Neutrosophic Logic (NL) proposed recently by mathematician L. Smarandache [19] is an 
extension or generalization of fuzzy logic. Conventional fuzzy logic represents a logical variable, 
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/ as ordered pair 1= (t,f) where t is considered the degree of truth and / is the degree of falsity, 
such that t+ f =1 . In neutrosophic logic, a logical variable x is described by an ordered triple 
composed of neutrosophic components, x= (t, i, f) where t, f are same as defined in fuzzy logic 
and i is the level of indeterminacy. In its most general form, each neutrosophic component can 
vary individually in the interval [0,1]. 

This generic definition thus allows neutrosophic logic to be able to deal with paradoxes, which 
are propositions that are true and false in the same time. Formally, a NL(paradox)=(l, I, 1). 
Fuzzy logic cannot do this because in fuzzy logic the sum of fuzzy components should be 1. 
However, to maintain consistency with fuzzy logic and probability, in our adaption of 
neutrosophic logic, it is assumed that 

t+f =1.0 (1) 

and the indeterminacy component is allowed to vary independently in interval [0,1]. Thus, 

t+i+f >=1.0 (2) 

The added indeterminacy component is useful to characterize imprecision of possessed 
knowledge which is common in AR. 

Combination of two or more neutrosophic values can be interpreted by logical connectives. For 
any two neutrosophic variables xi = (ti, ii, fi) and a?= (t 2 ,i 2 ,fi), the logical connectives not (~), 
conjunction (A) and disjunction (V) are defined as follows: 

if, x 3 = ~ xi then x 3 = (/j , i h ti) (3) 

if, x 3 = xi A X 2 then x 3 = ( min(ti, C), i = max(ii, i 2 ),f= max(fij 2 )) (4) 

if, x 3 = xj V X 2 then x 3 = ( max( t /_ I 2 ), i = max(ii, U), f= mintfify )) (5) 

Conjunction and disjunction connectives can be similarly defined for many neutrosophic 
variables as both max and min are group operators. As with fuzzy logic, falsity dominates in 
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conjunction and truthness in disjunction. Neutrosophic logic has been used to extend fuzzy 
classification in [20]. In this work, we make use of neutrosophic logic to extend our FCA based 
activity model. 

b. Neutrosophic Formal Concept Analysis 

Formal Concept Analysis has been used for AR in [6]. In this section, key definitions of formal 
concept analysis have been modified by including concepts of neutrosophic logic. 

Definition 1: A Neutrosophic formal context, N, is defined as a triplet} S,y, R], where S is a set 
of situations, y is the set of conceptually scaled contexts, and R is a relation in domain S xy. 
Each member (s, Yi) G R has a membership value, p, defined as a triplet { p t (s, Yi)> Pi(s, Yi) , 
pKs, Yi)]each in interval [0,1]. Each element of R represents, context ‘ Yi’ determining situation 
‘si’ is p t (s, Yi) true, Mi(s, Yi) indeterminate and pf(s, Yi) false. 



For AR, exact membership values have been calculated from the training dataset as per equations 
(6 - 8 ): 



fit(s, Yi) 



freq(s| Yi ) 
total_freq(s) 



(6) 



Pi(s, Yi) = max 



[ freqCYih 
(total_freq(s 



iU 

> 0 ) 



where ] G {S — s} 



(7) 



W (s, Yi ) = max V i e (y - Yi } (8) 

With respect to the training set, the frequency of context ‘m’ for a given activity, s, that is, freq(sl 
m) is the number of times the given value is present in that situation. The total_freq(s) is total 
instances of s, present in the training dataset. Another way of defining membership values for 
each instance in R is to obtain them from a domain expert or other mining approaches [21]. 
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Definition 2: Given a Neutrosophic formal context, N = {S, y, R}, AQ S and B £y and truth 
membership threshold, TM 



Neutrosophic extent, NE (B) = {s G Al V c G B: p t (s,c) — TM } 
Neutrosophic intent, NI (A) = {c6 ylVs£A:p t (s,c) > TM } 



(9) 

( 10 ) 



Definition 3: For a Neutrosophic formal context, N, a Neutrosophied Concept, NC is a pair (0 
(A), 0(B) ) where 0 (A) Q A and 0 (B) Q B such that, 

NI(0(A)) = 0(B) and NE(0(B))= 0(A) 

Vg £0 (A) has values of neutrosophic components as follows: 



p t = min (p t (g, c)) Vc £ 0(B) 



( 11 ) 



Pi= max (pi(g, c)) Vc £ 0(B) 



( 12 ) 



Pf= max (Pf(g,c)) Vc £ 0(B) 



(13) 



Given an intent set 0(B), p L , p , and p f represent truthness, indeterminacy and falsity values of the 
occurrence of each situation in the extent. Algorithms to extract and organize the set of 
neutrosophic concepts and organize them as neutrosophic concept lattice are given in next 
section. 



IV. NEUTROSOPHIC CONCEPT LATTICE BASED ACTIVITY MODEL AND 

ACTIVITY RECOGNITION 



In this section, creation of activity model based on neutrosophic concept lattice has been 
described. Model is constructed in two parts, first of these is the creation of list of all 
neutrosophic concepts. Second part is about arranging the concepts in form of lattice. 



1532 




INTERNATIONAL IOURNAL ON SMART SENSING AND INTELLIGENT SYSTEMS VOL. 8, NO. 3, SEPTEMBER 2015 



a. Neutrosophic Concept Lattice based Activity Model 

Neutrosophic concept lattice based model is obtained by creating a list of neutrosophic concepts 
and organizing them in lattice order. Algorithm 1 calculates all neutrosophic concepts. For 
implementation of the algorithm, few Abstract Data Types (ADTs) have been defined. 
neutro_mem has been defined to represent the membership triplet of an object. A generic object 
has been defined by the ADT Object characterized by its name and membership values. Attribute 
that possesses a name has also been defined as an ADT. Extent and Intent are another ADTs used 
to represent neutrosophic extents and intents as defined earlier. Using these ADTs, Concept data 
structure has been defined in terms of extent and intent. 



ADT neutro_mem { true_mem; indeterminacy_mem; false_mem, bal_score j 
ADT Object {Name, neutron_mem; } 

ADT Attribute {Name }; 

ADT Extent {Set of Objects;} 

ADT Intent { Set of Attributes; } 

ADT Concept {Extent; Intent;} 



Algorithm 1: Compute_Set_of_All_Concepts 
Input: Neutrosophic Context (O, y, R) 
Output: Set_All_Concepts 



Method: 

Const TM: truth membership threshold 

1. Set All Concepts := O 

2. New_Concept.Extent := Cal_Neutro_Extent(y) 

3. New_Concept.Intent := y; 

4. New_Concept. Extent.neutro_mem := Cal_Neutro_Mem (New_Concept.Extent , 
New_Concept.Intent) 

5. Set_All_Concepts := New_Concept U Set_All_Concepts 

6. for A c y : V a G A, p t (a) > TM 

7. temp_extent := Cal_Neutro_Extent (A) 

8. flag:=l 
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9. for each concept G Set_All_Concepts 

10. if concept. Extent = temp_extent 

11. flag:=0 

12. end if 

13. end for 

14. if (flag) 

15. New_Concept. Extent := temp_extent 

16. New_Concept. Intent := A 

17. New_Concept. Extent .neutro_mem := Cal_Neutro_Mem (New_Concept.Extent , 

Ne w_C oncept . Intent) 

18. Set_All_Concepts := New_Concept U Set_All_Concepts 

19. end if 

20. end for 

Function Cal_Neutro_Extent(A) 

Input: Subset of Attributes, A 

Output: Extent of A 
Method: 

Extent = O; 
for each a G A 

attribute_extent := O 
for each obj G O 

if (R[obj][a].true_mem > MemThreshold) 

attribute_extent := attribute_extent U obj 

end If 

end for 

Extent = Extent fl attribute_extent; 
end for 

Function Cal_Neutro_Mem () 

Input: X, Y : Concept.extent and Concept. Intent 
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Output: Neutro_Mem of Neutrosophic concept (X, Y) 

Method: 

for each xG X 

min_t= 1.0 // minimum truth value for all attributes of an object 

max_i=0.0 // maximum indeterminate value for all attributes of an object 

max_f = 0.0 // maximum false value for all attributes of an object 

for each y G Y 

if( R[x] [y] .true_mem < min_t) then 
min_t: = R[x][y].true_mem 
if(R[x][y].indeter_mem > max_i) 

max_i:= R[x][y].indeter_mem 
if (R[x] [y] .false_mem > max_f) 

max _f := R[x][y].false_mem; 



end for 

x. neutro_mem.true_mem := min_t; 
x.neutro_mem. indeterminacy_mem := max_i; 
x. neutro_mem. false_mem = max_f; 

return neutro_mem_bal= 0.5 * x. neutro_mem.true_mem - 0.5 * x.neutro_mem. 

indeterminacy_mem 



end for 



The relation R, has been implemented as a two dimensional array of size '’number of objects * 
number of attributes ' . Each element of array, R[i][j], is assigned membership triplet if T 
possesses f ’ else zero. 

Algorithm 1 takes as input neutrosophic formal context, described by set of objects, set of 
attributes and relation R. The algorithm then works as follows: 

Set of all concepts is initialized to be empty in line 1. 

In lines (2-5), the first concept is calculated as taking Intent as complete set of 
attributes and calculating its corresponding extent. It is added to overall concept set. 
This concept will act as the infimum to create a complete lattice [21]. 
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Lines (6 -20) describe generation of new concepts. Iterations to calculate all possible 
intents, find their extent and add to the set of all concepts, if it is already not there are 
defined. 

Line 6 each time generates a possible intent as proper subset of attributes such that 
each element has truth membership greater than threshold. Extent of generated intent 
is calculated in Line 7 and saved in a temporary extent. This is done as, it is yet to be 
decided whether a new concept has been found or not. 

In Lines (8 -13) temporary extent is compared with extents of existing concepts and a 
flag is reset if so found. No new concept is added in this case. 

Lines (14 -19) execute if concept has not been generated before. New concept is 
created by calculating neutrosophic membership of each object of extent using 
memberships of each attribute of intent. 

The algorithm ends with concept generation loop in line 20 
Algorithm 1 makes use of two functions. Cal_Neutro_Extent( ) takes as input neutrosophic intent 
and calculates neutrosophic extent as per definition 2. Cal_Neutro_Mem() calculates 
neutrosophic membership triples for each object in a given extent according to memberships of 
attribute in intent. 

For AR problem, objects are considered as activities and their attributes are relevant contexts. 
Both truthness and indeterminacy membership values have been given equal weights to decide 
membership of a context in an activity. The indeterminacy score penalizes the membership of a 
context in any activity. Computation of fuzzy lattice was also been done using same algorithm, 
but only true membership is utilized for extent calculation. 

In FCA, set of concepts are organized as lattice representing sub concept and concept relation 
[18]. However, here instead of lattice construction, it suffices to create a partially ordered list of 
concepts arranged in descending order of cardinality of extents of each concept. The complexity 
of computation of ordered set of extent in lexicographic order is 0(lnl“) while that of lattice 
construction is typically is O (Ini I A I (101 + I A I)) [22], where Ini is number of concepts, A is the 
set of attributes and O is the set of objects. The neutrosophic formal context, partial ordered list 
of concepts and neutrosophic membership of each extent forms our sought activity model that 
can be stored in suitable data structures for real time AR. 
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In the testing phase, the contexts at a given time instance, t, are used to infer activities. An 
activity query is denoted by A q = {Q,.. C, (. the set of current contexts. Query is input to 
Algorithm 2 and recognized activity is obtained as output. In case, the query context is not 
present in intent of any of the concepts, user will get an estimate of actual activity as most likely 
or probable. 

Algorithm 2 iterates over complete set of concepts in lines (1-9) to find exact match of activity 
corresponding to queried context set, Aq. 

Algorithm 2: Predict_Activity_Set 

Input: C:= Set_All_Concepts, Queried Context Description Aq:= { Ci,.. C, } 

Output: Recognized Activity 

Method: 

1. for each concept, c 

2. if Aq Q c. Intent then 

3. if |c.Extent|= 1 then ():= c. Extent 

4. else for each o in c. Extent 

5. find o such that, o.neutromem = max(c.Extent.neutromem) 

6. p:=o 

7. end if 

8. end if 

9. end for 

10. if p:= 0 

11. c’:= 0 

12. for all , c G C, find c such that max ( \ Aq n c lntent | s j 

V|Aq U c.Intent|/ 

13. c’-c’Uc 

14. end if 

15. p’ = n[ii c' t for all c'j G c’ 

16. p” = u£i c'i for all c' t G c’ 
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17. S(precise)= (3 //set of currently occurring situations 

18. S(most_likely) = (3’ //most likely set of situations 

19. S(probable) = (3” //as the probable set of situations 

20. End 

The algorithm works as follows: 

Lines (2-3) represent the simplest case when Aq is found as subset of intent of one of 
the concepts with single activity in its extent. Extent of this concept is output as 
recognized activity. 

Lines (4-8) consider the case when Aq is found as subset of an intent whose extent has 
more than one activity. The recognized activity is then considered as the one have 
maximum neutrosophic value. Activity that has maximum truthness and minimum 
indeterminacy membership in Aq is considered as one having maximum neutrosophic 
value. 

Lines 10 -16 are executed if Aq is not found in any of the concept intent. An 
approximate answer is sought then. In line 10-14, the list of concepts is traversed 
again to generate a set of concepts, c’, having concepts maximally similar to Aq. 
Similarity is computed by calculating Jaccard’s coefficient with respect to Aq and 
each intent. 

In lines 15-16, set of most likely is calculated as activities common in extents of c\ 
The set of probable activities are calculated as all possible activities present in any of 
the extents of c’. 

All sets of recognized activities are finally output in lines 17-19 and algorithm is 
terminated in line 20. 

The time complexity of Algorithm 2 is linear and computed as O (INI) in case of exact match and 
as O (2INI) in case of most probable and possible matches. In the next section, the proposed 
algorithms are evaluated on publicly available datasets. 
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V. RESULTS AND PERFORMANCE ANALYSIS 

AR model developed in earlier section have been validated on publicly available multi modal 
sensor based dataset called “OPPORTUINITY” [23]. It has been used by several researchers to 
examine their methods of activity recognition [24] . 

Database has multi-modal sensor data annotated with two types of ground truths recorded from 
four subjects in five run of each. One is low level actions and other is high level activities. Low 
level actions, as described in Table 1, have been used as contexts in which the actual activity 
takes place. Activities annotated in the dataset are described in Table 2. Table 2 also gives 
general description of how an activity was performed in the said environment. Numeric names 
used for contexts and each activity in concept set construction are enclosed in parenthesis in 
respective tables. 



Table 1: Activity Context, their Possible Values and Id of each Value 



Contexts 


Abstract Values 


Locomotion 


Null (1), Stand (2), Walk (3) , Sit (4), Lie (5) 


Left Arm 

Actions 


Null (6), Unlock (7), stir (8), lock (9), close (10), reach (11), open (12), sip 
(13), clean (14), bite (15), cut (16), spread (17), release (18), move (19) 


Right Arm 

Actions 


Null (44), Unlock (45), stir (46), lock (47), close (48), reach (49), open (50), 
sip (51), clean (52), bite (53), cut (54), spread (55), release (56), move (57) 


Object Used 

with Left Arm 


Null (20), Bottle (21), Salami (22), Bread (23), Sugar (24), Dishwasher (25), 
Switch (26), Milk (27), Lower Drawer (28), Spoon (29), Knife for cheese (30), 
Middle Drawer (31), Table (32), Glass (33), Cheese (34), Chair (35), Doorl 
(36), Door2 (37), Plate (38), Top Drawer (39), Fridge (40), Cup (41), Knife 
for salami (42), Lazy chair (43) 


Object Used 

with Right Arm 


Null (58), Bottle (59), Salami (60), Bread (61), Sugar (62), Dishwasher (63), 
Switch (64), Milk (65), Lower Drawer (66), Spoon (67), Knife for cheese (68), 
Middle Drawer (69), Table (70), Glass (71), Cheese (72), Chair (73), Doorl 
(74), Door2 (75), Plate (76), Top Drawer (77), Fridge (78), Cup (79), Knife 
for salami (80), Lazy chair (81) 
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Table 2: Activities Done and Their Descriptions 



Activity 
(Activity Id) 


General Process of Doing Activity 


Idle (0) 


No activity 


Relaxing (101) 


Close doors, Lie down and no other action or object usage 


Coffee time(102) 


Prepare Coffee using utensils like cup, spoon etc., ingredients like milk 
, sugar, appliance like Coffee Machine and accessing drawers, Subject is 
standing most of the time and walks to fetch things 


Early Morning 
(103) 


Goes out of the room for a walk, walks around in the room, standing 
occasionally to use some objects 


Cleanup (104) 


Cleans the table, put soiled dishes in dishwasher , put remaining food in 
fridge and drawers 


Sandwich Time 
(105) 


Fetch ingredients cheese, salami , bread etc. from drawers and fridge, sit on 
table , prepare and eat sandwich 



On observing the datasets it was found that when annotated activity was idle random values of 
contexts were occurring. Thus any instance of activity “Idle” was not used in training and testing 
of activity model. 

a. Learnt Activity Model and its Interpretation 

Activity Models were created for each subject using data of four runs for training the model and 
fifth run for testing. Twenty experiments including five for each of the four subjects using each 
run at a time for testing and remaining for training were performed. Figure 1 represents the 
concept lattice obtained by using first four runs of Subject 1 as training data. 

Lattice Navigator software has been utilized to visualize the lattice in Figure 1 [25]. Each node 
has been labelled using reduced labelling [22]. Lattice has two extreme concepts. The topmost 
concept represents the context values that are found in all activities. The bottommost concept 
represents the activity which has all possible contexts values in its description. All lattices 
generated with given datasets had empty extent for these concepts. Twenty lattice models were 
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obtained for testing the models. Results presented in next section are average performance over 
all these models. 

The lattice is stored in memory as graph data structure and accessed in breadth first manner for 
context query matching using Algorithm 2. In case the matched concept has an extent with more 
than one activity, recognition of activity is resolved by maximizing neutrosophic membership. 

For example, on breadth first traversal of lattice, a queried context description, 

A q = {Locomotion: Stand (2); Left Arm Actions: Reach (II)', Right Arm Actions: Lower Drawer 
(28); Object Used with Left Arm: Null (44); Object Used with Right Arm: Null (58)} matches the 
6 th node in 4 th level of lattice with extent {Coffee Time and Sandwich Time}. The actual activity 
is recognized by computing neutrosophic score for each activity and output the one with higher 
score. 

Validation of activity models was done by testing with activity run data of same subject for 
which the model was trained as well as for other subjects. For establishing the advantage of 
proposed approach, performance of neutrosophic concept lattice model has been compared 
against conventional fuzzy concept lattice based activity model. 
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{{Cleanup, 
Coffee Time, 
Sandwich 
Time}, {10, 
11, 12, 18, 2, 
20, 3, 40, 44, 
48, 49, 50, 
56, 58, 6, 66, 
78}} 



{{Cleanup, 
Coffee Time, 
Early 
Morning, 
Sandwich 
Time}, {10, 
11, 12, 18, 2, 
20, 3, 40, 44, 
48, 49, 50, 
56, 58, 6, 
78}} 



{{Cleanup, 
Coffee Time, 
Early Morning, 
Relaxing, 
Sandwich 
Time}, {2, 20, 
3, 44, 58, 6}} 



{{Cleanup, 

Early 

Morning}, {10, 
11, 12, 18, 19, 
2, 20, 25, 3, 
4, 40, 41, 44, 
48, 49, 50, 56, 
57, 58, 6, 64, 
73, 75, 78, 
79}} 



{{Cleanup, 
Coffee Time}, 
{ 10 , 11 , 12 , 
18, 2, 20, 25, 
26, 3, 40, 44, 
48, 49, 50, 
56, 58, 6, 63, 
64, 66, 74, 
75, 78}} 



{{Cleanup} 1 ; 
{ 10 , 11 , 12 , 
14, 18, 19, 2, 
20, 22, 23, 
25, 26, 3, 30, 
32, 33, 34, 
35, 36, 38, 4, 
40, 41, 42, 
44, 48, 49, 
50, 52, 56, 
57, 58, 6, 60, 
61, 63, 64, 
66, 68, 70, 
71, 72, 73, 
74, 75, 76, 
78, 79, 80}} 



{{Cleanup, 
Early 
Morning, 
Sandwich 
Time}, {10, 
11, 12, 18, 
19, 2, 20, 3, 
4, 40, 41, 44,. 

48, 49, 50, 
56, 57, 58, 

73, 78, 79 




{{Coffee 
Time, Early 
Morning}, {10, 
11, 12, 18, 2, 
20, 25, 3, 40, 
44, 48, 49, 50, 
56, 58, 6, 64, 
75, 77, 78}} 



{{Cleanup, 
Early 
Morning, 
Relaxing, 
Sandwich 
Time}, {2, 20, 
3, 4, 44, 58, 
6 }} 



{{Cleanup, 
Sandwich 
Time}, {10, 
,11, 12, 18, 
1 ^ 20 , 22 , 
23, 3/51V33, 
34, 35, 38, 
40, 41, 42, 
44, 48, 49, 
50, 56, 57, 
58, 6, 60, 61, 
66, 68, 71, 
72, 73, 76, 
78, 79, 80}} 



{{Cleanup, 
Coffee Time, 
Early 

Morning}, {10, 
11, 12, 18, 2, 
20, 25, 3, 40, 
44, 48, 49, 50, 
56, 58, 6, 64, 
75, 78}} 



{{Coffee 
Time, 
Sandwich 
Time},{l, 10, 
11, 12, 18, 2, 
20, 28, 3, 31, 
39, 40, 44, 
48, 49, 50, 
56, 58, 6, 66, 
69, 77, 78}} 







{{Coffee 


{{Coffee 


Time, Early 


Time},{l, 10, 




Morning, 


11, 12, 18, 2, 




Sandwich 


20, 25, 26, 


Time}, {10, 


28, 3, 31, 39, 


11, 12, 18, 2, 


40, 44, 45, 


20, 3, 40, 44, 


48, 49, 50, 




48, 49, 50, 


56, 58, 6, 63, 


56, 58, 6, 77, 


64, 66, 69, 




78}} 


74, 75, 77, 


X. 




\ 78}} 


{{Relaxinq, 


X 




Sandwich 




\ 


Time},{l, 2, 


{{Coffee 




20, 3, 4, 44, 


Time, 




58, 6}} 


Relaxing, 






Sandwich 


{{Sandwich 




Time},{l, 


Time},{l, 10, 




20, 3, 44, 


11, 12, 13, 


^-{{ftelaxing}, 


6}} 


15, 18, 19, 2^ 




20, 


{1, 2, 20, 3, 




23^2873, 30, 


4, 44, 5, 58, 




<31, 33, 34, 


6}} 





Time}, {10, 
11, 12, 18, 
19, 2, 20, 3, 
4, 40, 41, 44, 
48, 49, 50, 
Sly 56, 57, 
58, 6, ^65, 73, 
77, 78, 79}} 



35, 38, 39, 4, 
40, 41, 42, 
44, 48, 49, 
50, 51, 53, 
54, 55, 56, 
57, 58, 59, 6, 
60, 61, 65, 
66, 68, 69, 
71, 72, 73, 
76, 77, 78, 
79, 80}} 



{{},{!, 10 , 
11, 12, 13, 
14, 15, 16, 
17, 18, 19, 2, 
20 , 21 , 22 , 
23, 24, 25, 
26, 27, 28, 
29, 3, 30, 31, 
32, 33, 34, 
35, 36, 37, 
38, 39, 4, 40, 
41, 42, 43, 
44, 45, 46, 
47, 48, 49, 5, 
50, 51, 52, 
53, 54, 55, 
56, 57, 58, 
59, 6, 60, 61, 
62, 63, 64, 
65, 66, 67, 
68, 69, 7, 70, 
71, 72, 73, 
74, 75, 76, 
77, 78, 79, 8, 
80, 81, 9}} 



Figure 1: Lattice Based Activity Model Learnt from data of Subject 1 (Visualized in Lattice 

Navigator) 
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b. Validation of Activity Recognition on Same Subject 
Experiments without Null Contextual Information: 

As can be seen in Table 1, each context has been labelled as ‘NULL ’ at times. This value doesn’t 
have any clear cut meaning in dataset. Thus, in the first experiment all data instances where value 
of even one of the contexts was ‘NULL ’ were discarded. Confusion matrix of Table 3 shows 
recognition results when 80% data of Subject 1 was used for training and remaining 20% for 
testing. Activity recognition was very encouraging with 93.3% accuracy obtained. These 
instances represent best quality data of dataset. It can be observed from Table 3 that only 
contextual information obtained from sensors embedded in objects is presenting confusing results 
as same objects are touched in many activities. 

A major concern here is complete elimination of ‘ Relaxing ’ and partial elimination of ‘ Early 
Morning ’ activities from the dataset. 

Table 3: Confusion Matrix Obtained By Ignoring All Null Values of Contexts 





Predicted Situations 


Relaxing 


Coffee Time 


Early Morning 


Cleanup 


Sandwich Time 


Actual Situations 


Relaxing 


0 


0 


0 


0 


0 


Coffee time 


0 


701 


0 


28 


0 


Early morning 


0 


0 


45 


0 


0 


Cleanup 


0 


0 


22 


1025 


44 


Sandwich Time 


0 


0 


191 


55 


1501 



This is due to the fact that in relaxing activity once the subject lies down, values of all other 
contexts becomes null as he neither does any action nor uses any object. Similarly in “Early 
Morning” activity the subject goes out of room for walk, thus making all other contextual values 
as ‘NULL’. 

Experiments with Null Contextual Information: 

Due to the problem of elimination of two activities from activity model, the next set of 
experiments considered ‘Null ’ as possible context values for each type of contexts. Activity 
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model was trained with conventional fuzzy lattice method and proposed neutrosophic lattice 
method. 




Figure 2: Mean accuracy of classification over five runs of each subject 



Models were trained and tested five times for each subject by taking four of the runs as training 
data and remaining for testing. Subject wise mean accuracy of recognition of each activity was 
evaluated for each run and has been presented in Figure 2. 

It can be seen that neutrosophic lattice performs at par with activity recognition of fuzzy lattice. 
Inclusion of indeterminacy component in neutrosophic logic has increased its semantics while not 
affecting the recognition accuracy. Dataset is imbalanced in terms of number of occurrences of 
each activity. Thus, validation results were analysed for activity wise recognition rate of each run 
of each subject. 

Table 4: Confusion Matrix of one Runs of Subject 3 





Predicted Activities 


Relaxing 


Coffee Time 


Early Morning 


Cleanup 


Sandwich Time 


Actual Situations 


Relaxing 


822 


0 


2 


255 


404 


Coffee time 


0 


2004 


205 


473 


103 


Early morning 


0 


459 


4431 


2117 


0 


Cleanup 


0 


327 


103 


3032 


542 


Sandwich Time 


0 


648 


342 


1420 


8175 
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In Table 4, one of the confusion matrices obtained while testing a run of subject 3 has been 
shown. Inclusion of null values for each type of context has increased spill overs to neighbouring 
activities. This is because each context’s values are ‘NULL ’ in many activities. 

Other observation from this confusion matrix is about the Cleanup activity (Recall rate: 41.5%). 
This activity is often mistaken as coffee time, early morning and sandwich time. It is the usage of 
dishwasher that separates Cleanup from other three activities. Thus, only those instances of 
cleanup where usage of dishwasher is made are correctly recognized, while others are recognized 
to other activities depending on the neutrosophic membership value of that activity for a given 
test data. 

In Figure 3, mean recall rate of each activity has been shown. Recognition of Sandwich time and 
Early Morning activity is improved greatly in our method due to their lopsided truth membership 
values and less indeterminacy. Locomotion context with ‘sit’ value is unique and most common 
to Sandwich Time activity. Similarly, ‘ Null ’ values for locomotion context has high truth 
membership in Early Morning activity. In both cases, the use of indeterminacy factor enhances 
the recognition rate of these activities. 



□ Subject 1 ■ Subject 2 ■ Subject 3 ■ Subject 4 
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Figure 3: Activity wise mean recall over five runs of each subject 
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A loss in recall of indeterminate activities like ‘ Cleanup ’ and ‘ Coffee Time ’ and gain in recall of 
distinguishable activities like Sandwich Time and Early Morning proves the efficacy of approach 
in achieving its objective of highlighting non-deterministic activities. 



Subject 1 ■ Subject 2 ■ Subject 3 ■ Subject 4 



0J 

1- 

Q- 
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Neutro 


Fuzzy 


Neutro 


Fuzzy 


Neutro 


Fuzzy 


Neutro 


Fuzzy 


Neutro 


Relaxing 


Coffee time 


Early morning 


Cleanup 


Sandwich 

Time 



Figure 4: Activity wise mean Precision over five runs of each subject 



Recall gave an idea of how many times each activity was correctly recognized. Next set of 
analysis over obtained results was to find precision of each class of activity. Precision is 
important for getting information on exactness or quality of recognition of each class (activity 
here) in imbalanced datasets. 

In Figure 4, activity wise precision of each activity for all four subjects has been reproduced. For 
Relaxing activity, precision is perfect hundred except for one case using neutrosophic logic. For 
understanding this result, raw data of subject 2 was examined closely for this activity particularly. 
In Relaxing activity, a subject typically lies down on the lazy chair for most of the duration. It 
was found that Subject 2 was sitting most of the time during one of the runs of Relaxing. Sitting, 
however, is typical to Sandwich Time activity, thus the proposed neutrosophic lattice captured 
this as indeterminacy and low precision was observed for this case. It can be noted that fuzzy 
lattice could not capture this anomaly in training data. 

Similarly, for Sandwich Time which is otherwise deterministic activity, significant drop in 
precision was noticed for Subject 1 and 3. It was again explainable through raw data of these 
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subjects where during more than one runs these subjects have interleaved coffee sipping and 
cleaning with Sandwich Time activity. Thus, using our lattice, the non-determinism in the way an 
activity is carried has been identified well, which was not possible using fuzzy lattice approach. 
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Fuzzy 


Neutro 


Fuzzy 


Neutro 


Fuzzy 


Neutro 


Relaxing 


Coffee time 


Early morning 


Cleanup 


Sandwich 

Time 



Figure 5: Overall Activity Wise F Measure for Our Approach and Fuzzy Lattice 



Overall F-measure analysis of all subjects has been represented in Figure 5. As expected the F- 
measure for non- deterministic Cleanup activity is low. However, for other such activity that is 
Coffee Time, F measure is comparable to other activities due to improved precision of this 
activity for subjects 1, 3 and 4. Low precision of Subject 2 for Relaxing activity has contributed 
to reduced F measure of this activity. 

There were only about 0.5% instances where the test data did not match the intent of any of the 
concepts, thus calculation of most likely and probable activities for these were dropped. 



c. VALIDATION OF ACTIVITY RECOGNITION ACROSS SUBJECTS 



In activity recognition, it is important to analyse the validity of learnt model when applied over 
different users. For this analysis, datasets of all runs of one subject was used for training and 
tested on dataset of other subjects one by one. One of the results where activity model was 
trained on subject 1 and tested on all other subjects has been shown in Figure 6. 
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Subject 2 ■ Subject 3 ■ Subject 4 
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Figure 6: Activity Wise Mean Precision on Applying Activity Model across Subjects 




Figure 7: Subject Wise Accuracy over application of Models across Subjects 

Average precision has decreased as compared to fuzzy approach. However, for deterministic 
activities, neutrosophic method gave far better results. However, very low precision of Cleanup 
and Coffee Time is a concern and thus sensor sources for these activities need to be enhanced 
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urgently. It can be concluded that neutrosophic lattice based classifier helps in addressing 
ambiguity of interpretation of activities by highlighting them due to significantly reduced 
recognition (down to 10% in some cases). 

In Figure 7, overall accuracy of activity recognition obtained, when activity models learnt using 
Subject l’s data are tested over other subjects’ data is shown. The overall accuracy indicates 
consistent performance of the model. It can be thus concluded that if activity wise precision 
issues are handled well, lattice based models can withstand variations of activity dissemination 
by different persons. Thus, need for training the model for every new user is eliminated. This is a 
big advantage while designing ready to use sensor based activity recognition solution. 

d. Comparison of Activity Recognition with Other Approaches 
Study of effectiveness of context data in describing activities on same dataset was also found in 
[26]. Purpose of evaluation was to select the relevant set of contexts. J48 decision tree, Hidden 
Naive Bayes and simple instance based learner IBK classifiers were used in that study. The paper 
did not clarify about which subset of dataset was taken in their study. Therefore, their methods 
could not be verified by us on our own and evaluation results mentioned in their paper have been 
used directly. The comparison results of their reported accuracy and against mean performance 
of proposed method is shown in Figure 8. 
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Figure 8: Evaluation of Proposed Model against results published in [26] 
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The accuracy of deterministic activities Relaxing , Early Morning and Sandwich Time is higher or 
equivalent to machine learning based approaches. The accuracy of non-deterministic approaches 
is slightly lesser, as our approach penalizes the recognition in such activities. The recognition 
algorithms used in [26] work like a black box and recognition process is not explainable. In our 
case the recognition process is semantically clear and self-explanatory. 

While strength of proposed model is in identifying non determinism, it has few limitations 
also. It is based on similarity computation to recognize activities and cannot perform reasoning to 
derive new information such as those supported by ontology based activity models. Other 
shortcoming is its inability to only highlight and not improve performance of non-deterministic 
activity. The proposed method can’t differentiate between non-determinism due to non - 
availability of contextual information from non- determinism caused due to interleaving of 
otherwise deterministic activities. These shortcomings can be worked upon by further enhancing 
the proposed method. 

For large scale usage, AR methods across similar homes in a building can be investigated. One 
such study with respect to scalability of home monitoring systems, determining wellness through 
AR, to smart buildings has been presented in [27]. 



VI. CONCLUSIONS 

In this paper, human activity recognition from sensor derived low level abstract information that 
is “contexts” has been studied. Existing approaches for solving this problem have shown good 
results for recognizing simple actions. Recognition of high level activities defined by certain 
sequence of contexts and taking place at different times or place has also been attempted. 
However, humans do activities in their own manner and seldom follow known sequence while 
doing certain activity. Activities taking place at same place within same time frame and having 
similar context values can become non-deterministic if at least one distinguishing context value is 
not present in it. Such activities give poor recognition results with almost all existing approaches. 
In this paper, Neutrosophic logic, a generalization of fuzzy logic has been utilized to quantify 
non-determinism and include it in Formal Concept Analysis based activity model. The proposed 
method does not increase the accuracy of recognition of non-deterministic activities but it works 
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towards highlighting them during recognition. This is done by calculating non-determinism 
associated with contextual description of activities and utilizing it as penalizing factor in 
recognition. Thus, recognition performance of non-deterministic activities gets significantly 
reduced while is increased for deterministic activities. The algorithms were tested on benchmark 
context and activity datasets. Difference in recognition performance of deterministic and non- 
deterministic activities was not significant in other approaches. The proposed approach also 
successfully revealed anomalies in input data which was not apparent in other methods. 

It is thus concluded that our approach can be effective for testing correctness of context based 
activity determination before on site installations. New relevant contexts can be added for 
activities which are not recognizable with current set of contexts. This may require deployment of 
more number or types of sensors. 
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